System for surface analysis and method thereof

ABSTRACT

A system and method for analyzing a surface of an object is provided. The system includes a 3D measurement device operable to acquire a plurality of points on the surface of the object and determine 3D coordinates for each of the points. The system further includes processors operably coupled to the 3D measurement device. The processors are responsive to computer instructions when executed on the processors for performing a method comprising: generating a point cloud from the 3D coordinates of the plurality of points; extracting a first set of points from the plurality of points; defining a first reference geometry through the first set of points; measuring at least one first metric from each of the points in the first set of points to the first reference geometry; and identifying a nonconforming feature based at least in part on the at least one first metric.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/589,126, filed Nov. 21, 2017, the entire disclosure of which isincorporated herein by reference.

BACKGROUND

The present application is directed to a system analyzing a surface ofan object, and in particular for analyzing a surface without access to acomputer-aided-design (CAD) model.

A 3D laser scanner time-of-flight (TOF) coordinate measurement device isa type of metrology device that is used for determining coordinates ofsurfaces of an object. A 3D laser scanner of this type steers a beam oflight to a non-cooperative target such as a diffusely scattering surfaceof an object. A distance meter in the device measures a distance to theobject, and angular encoders measure the angles of rotation of two axlesin the device. The measured distance and two angles enable a processorin the device to determine the 3D coordinates of the target.

A TOF laser scanner is a scanner in which the distance to a target pointis determined based on the speed of light in air between the scanner anda target point. Laser scanners are typically used for scanning closed oropen spaces such as interior areas of buildings, industrialinstallations and tunnels. They may be used, for example, in industrialapplications and accident reconstruction applications. A laser scanneroptically scans and measures objects in a volume around the scannerthrough the acquisition of data points representing object surfaceswithin the volume. Such data points are obtained by transmitting a beamof light onto the objects and collecting the reflected or scatteredlight to determine the distance, two-angles (i.e., an azimuth and azenith angle), and optionally a gray-scale value. This raw scan data iscollected, stored and sent to a processor or processors to generate a 3Dimage representing the scanned area or object.

Generating an image requires at least three values for each data point.These three values may include the distance and two angles, or may betransformed values, such as the x, y, z coordinates. In an embodiment,an image is also based on a fourth gray-scale value, which is a valuerelated to irradiance of scattered light returning to the scanner.

Most TOF scanners direct the beam of light within the measurement volumeby steering the light with a beam steering mechanism. The beam steeringmechanism includes a first motor that steers the beam of light about afirst axis by a first angle that is measured by a first angular encoder(or other angle transducer). The beam steering mechanism also includes asecond motor that steers the beam of light about a second axis by asecond angle that is measured by a second angular encoder (or otherangle transducer).

Many contemporary laser scanners include a camera mounted on the laserscanner for gathering camera digital images of the environment and forpresenting the camera digital images to an operator of the laserscanner. By viewing the camera images, the operator of the scanner candetermine the field of view of the measured volume and adjust settingson the laser scanner to measure over a larger or smaller region ofspace. In addition, the camera digital images may be transmitted to aprocessor to add color to the scanner image. To generate a color scannerimage, at least three positional coordinates (such as x, y, z) and threecolor values (such as red, green, blue “RGB”) are collected for eachdata point.

A 3D image of a scene may require multiple scans from differentregistration positions. The overlapping scans are registered in a jointcoordinate system. Such registration is performed by matching targets inoverlapping regions of the multiple scans. The targets may be artificialtargets such as spheres or checkerboards or they may be natural featuressuch as corners or edges of walls. Some registration procedures involverelatively time-consuming manual procedures such as identifying by auser each target and matching the targets obtained by the scanner ineach of the different registration positions. Some registrationprocedures also require establishing an external “control network” ofregistration targets measured by an external device such as a totalstation.

In some applications, once an object is scanned and the registrationcompleted, the point cloud is then analyzed or inspected to determine ifparameters are out of specification or are otherwise nonconforming. Itshould be appreciated that manually analyzing the point cloud could be aslow and tedious process. Attempts at automating the analysis of thepoint cloud typically focus on comparing the point cloud to anelectronic model of the object being measured, such as acomputer-aided-design (CAD) model. First the CAD model and the pointcloud are aligned. Next, a comparison of the point cloud to the CADmodel is performed and deviations are highlighted. In some instances,the deviations are noted based on a predetermined tolerance. In stillother instances predetermined attributes, such as dimensions forexample, are identified by the operator prior to the comparison.

While the use of a CAD model for comparison with the point cloud allowsfor some inspection of the object being measured, CAD models for objectsare not always available. For example, when the measurements are beingperformed by a third party, in other words someone other than themanufacturer of the object, the proprietary CAD model may not beavailable. Third parties that may be interested in measuring objectsinclude insurance companies looking to estimate damage or airlines ininspecting airplanes for example.

Accordingly, while existing systems for measuring and analyzing surfacesare suitable for their intended purposes, what is needed is a systemhaving certain features of embodiments of the present invention.

BRIEF DESCRIPTION

According to one aspect of the invention, a system for analyzing asurface of an object is provided. The system includes athree-dimensional (3D) measurement device operable to acquire aplurality of points on the surface of the object and determine 3Dcoordinates for each of the plurality of points. The system furtherincludes one or more processors operably coupled to the 3D measurementdevice. The one or more processors are responsive to executable computerinstructions when executed on the one or more processors for performinga method comprising: generating a point cloud from the 3D coordinates ofthe plurality of points; extracting a first set of points from theplurality of points; defining a first reference geometry through thefirst set of points; measuring at least one first metric from each ofthe points in the first set of points to the first reference geometry;and identifying a nonconforming feature based at least in part on the atleast one first metric.

In addition to one or more of the features described herein, or as analternative, further embodiments of the system may include the methodtransmitting a signal when the distance exceeds a predeterminedthreshold. In addition to one or more of the features described herein,or as an alternative, further embodiments of the system may include themethod segmenting surfaces in the point cloud into a set of surfacesbased on edges of the object. In addition to one or more of the featuresdescribed herein, or as an alternative, further embodiments of thesystem may include the method defining an area of interest, the firstset of points being disposed within the area of interest, and moving thearea of interest over each surface in the set of surfaces, wherein foreach movement of the area of interest a second set of points isextracted from the plurality of points, a second reference geometry isfit through the second set of points, a second metric is measured fromeach of the points in the set of points to the reference geometry, and anonconforming feature is identified based at least in part on the secondmetric.

In addition to one or more of the features described herein, or as analternative, further embodiments of the system may include the firstreference geometry being a first plane, and the second referencegeometry being a second plane. In addition to one or more of thefeatures described herein, or as an alternative, further embodiments ofthe system may include the at least one first metric being a firstdistance from the first plane to each of the points in the set ofpoints, and the second metric being a second distance from the secondplane to each of the points in the set of points. In addition to one ormore of the features described herein, or as an alternative, furtherembodiments of the system may include the first distance and seconddistance being determined along a vector normal to the first plane andthe second plane.

In addition to one or more of the features described herein, or as analternative, further embodiments of the system may include a mobileplatform, wherein the 3D measurement device is coupled to the mobileplatform. In addition to one or more of the features described herein,or as an alternative, further embodiments of the system may include themobile platform being operable to autonomously move along apredetermined path.

In addition to one or more of the features described herein, or as analternative, further embodiments of the system may include a mobilecomputing device, the mobile computing device having one or more secondprocessors responsive to executable computer instructions when executedon the one or more second processors, the one or more second processorsbeing responsive to execute a method comprising: displaying a graphicalrepresentation of the object on a user interface; identifying locationsrelative to the object to performs measurements with the 3D measurementdevice in response to an input from an operator; and transmitting thelocations to the one or more processors.

In addition to one or more of the features described herein, or as analternative, further embodiments of the system may include the mobilecomputing device further having a 3D camera operable to measure a depthto a surface from the mobile computing device. In addition to one ormore of the features described herein, or as an alternative, furtherembodiments of the system may include the one or more second processorsare further responsive to execute a method comprising: acquiring aplurality of images of the object with the 3D camera; and generating thegraphical representation of the object based at least in part on theplurality of images. In addition to one or more of the featuresdescribed herein, or as an alternative, further embodiments of thesystem may include the nonconforming feature being at least one of adent, a bump, or a missing fastener.

In a further aspect of the invention, a method of analyzing a surface ofan object is provided. The method includes performing a scan of theobject with a 3D measurement device, the 3D measurement device beingoperable to measure 3D coordinates for a plurality of points on thesurface. A point cloud is generated from the 3D coordinates of theplurality of points. A first set of points are extracted from theplurality of points. A first reference geometry is defined through thefirst set of points. At least one first metric is measured from each ofthe points in the first set of points to the first reference geometry. Anonconforming feature is identified based at least in part on the atleast one first metric.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include segmentingsurfaces in the point cloud into a set of surfaces based on edges of theobject. In addition to one or more of the features described herein, oras an alternative, further embodiments of the method may includedefining an area of interest, the first set of points being disposedwithin the area of interest. In addition to one or more of the featuresdescribed herein, or as an alternative, further embodiments of themethod may include moving the area of interest over each surface in theset of surfaces, wherein for each movement of the area of interest asecond set of points is extracted from the plurality of points, a secondreference geometry is fit through the second set of points, a secondmetric is measured from each of the points in the set of points to thereference geometry, and a nonconforming feature is identified based atleast in part on the second metric.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include moving the 3Dmeasurement device about the object, and performing a plurality of scansas the 3D measurement device is moved about the object. In addition toone or more of the features described herein, or as an alternative,further embodiments of the method may include the 3D measurement devicebeing moved by an autonomous mobile platform. In addition to one or moreof the features described herein, or as an alternative, furtherembodiments of the method may include displaying a graphicalrepresentation of the object on a user interface, identifying locationsto perform each of the plurality of scans relative to the object inresponse to an input from an operator, and transmitting the locations tothe autonomous mobile platform.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include acquiring aplurality of images using a 3D camera, each of the plurality of imagesincluding depth information, and generating the graphical representationof the object based at least in part on the plurality of images. Inaddition to one or more of the features described herein, or as analternative, further embodiments of the method may include theidentified the nonconforming feature being at least one of a dent, abump or a missing fastener. In addition to one or more of the featuresdescribed herein, or as an alternative, further embodiments of themethod may include displaying the point cloud on the user interface, andchanging a color of the point cloud at a location of the nonconformingfeature.

In addition to one or more of the features described herein, or as analternative, further embodiments of the method may include the firstreference geometry is a plane, and the at least one first metric is adistance from the plane to from each of the points in the first set ofpoints. In addition to one or more of the features described herein, oras an alternative, further embodiments of the method may include thedistance being determined along a vector normal to the plane.

These and other advantages and features will become more apparent fromthe following description taken in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter, which is regarded as the invention, is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 is a schematic illustration of a system for analyzing surfaces inaccordance with an embodiment of the invention;

FIG. 2 is a perspective view of an autonomous mobile three-dimensional(3D) measurement device according to an embodiment;

FIG. 3 is a partial top perspective view of the mobile 3D measurementdevice of FIG. 2 according to an embodiment;

FIG. 4 a front perspective view of the mobile 3D measurement device ofFIG. 2 according to an embodiment;

FIG. 5 is an autonomous mobile 3D measurement device according toanother embodiment;

FIG. 6, FIG. 7 and FIG. 8 illustrate a laser scanner device for use withthe mobile 3D measurement device of FIG. 2 according to an embodiment;

FIG. 8 illustrates a schematic block diagram of the system of FIG. 1according to an embodiment;

FIG. 10 and FIG. 11 illustrate a user interface for a mobile computingdevice used in the system of FIG. 1 according to an embodiment;

FIG. 12 is an illustration of a plane fit through an area of interestaccording to an embodiment;

FIG. 13 and FIG. 14 are an illustration of a portion of the area ofinterest of FIG. 12 according to an embodiment;

FIG. 15 is an illustration of a user interface illustrating an area ofinterest on an object being measured according to an embodiment;

FIG. 16 is another illustration of a user interface illustrating an areaof interest on an object being measured according to an embodiment;

FIG. 17 is an illustration of a point cloud of the area of interest onthe object according to an embodiment; and

FIG. 18 illustrates a flow diagram of a method of operating the systemof FIG. 1.

The detailed description explains embodiments of the invention, togetherwith advantages and features, by way of example with reference to thedrawings.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention relates to a system for analyzing asurface of an object. Some embodiments provide advantages in providing asystem that allows for analyzing the surface of an object of damage ornonconforming fabrication using a 3D measurement device mounted on anautonomous mobile platform. Further embodiments provide advantages inproviding a system that allows for the analyzing the surface of anobject of damage or nonconforming fabrication without comparison to anelectronic model, such as a CAD model.

Referring now to FIG. 1, an embodiment is shown of a system 20 foranalyzing surfaces of an object, such as airplane 22 for example. Itshould be appreciated that while embodiments herein may refer to theobject as being an airplane 22, this is for exemplary purposes. In otherembodiments, other objects may be inspected, such as but not limited toautomobiles, trucks, boats, ships, and bridges for example. In theexemplary embodiment, the system 20 includes a mobile 3D measurementsystem 24 that is comprised of a mobile platform 26 and an integral 3Dmeasurement device 28. As will be discussed in more detail herein, themobile platform 26 may be an autonomously operated vehicle that movesalong a predetermined route to perform a plurality of scans. The 3Dcoordinates of points on the surface from these plurality of scans maybe used to generate a point cloud of the object.

It should be appreciated that while the exemplary embodiment describesthe surface analysis with respect to a measurement system 24 having amobile platform 26, this is for exemplary purposes and the claims shouldnot be so limited. In other embodiments, the surface analysis may beperformed from a single scan by a 3D measurement system, such as a laserscanner for example, that is performed at a single location.

Referring now to FIGS. 2-5 and FIG. 9, an embodiment is shown of the 3Dmeasurement system 24. In this embodiment, the mobile platform 26includes a body 30 and a plurality of wheels 32. The wheels 32 aredriven by one or more motors 34. A steering mechanism 36 is coupled toat least two of the wheels 32 to allow the changing of direction ofmobile platform 26. Mounted on the body 30 are a plurality of sensors38, 40, 42 that provide input to a controller or processor 44. In anembodiment, the sensor input is used by the processor 44 to identifyobstacles in the area about the mobile platform 26 and allow theprocessor 44 to change the speed or direction of the mobile platform 26to avoid contact with an obstacle or a person. In an embodiment, theprocessor 44, controls the motor 34 and steering mechanism 36 and canoperate autonomously along a predetermined route.

Processor 44 is a suitable electronic device capable of accepting dataand instructions, executing the instructions to process the data.Controller 44 may accept instructions through user interface, or throughother means such as but not limited to electronic data card, voiceactivation means, manually-operable selection and control means,radiated wavelength and electronic or electrical transfer.

In general, processor 44 accepts data from sensors 38, 40, 42 and isgiven certain instructions for the purpose of comparing the data fromsensors 38, 40, 42 to predetermined operational parameters. Processor 44provides operating signals to the motor 34 and steering mechanism 36.Processor 44 also accepts data from sensors 38, 40, 42, indicating, forexample, whether an obstacle. The controller 38 compares the operationalparameters to predetermined variances (e.g. probability of contact) andif the predetermined variance is exceeded, generates a signal that maybe used to indicate an alarm to an operator or the computer network 42.Additionally, the signal may initiate other control methods that adaptthe operation of the 3D measurement system 24 such as changing thedirection or speed of the mobile platform 26 to compensate for the outof variance operating parameter. The signal may also initiate othercontrol methods that initiate operation of the 3D measurement device 28when a predetermined location has been reached.

Processor 44 may also be coupled to external computer networks such as alocal area network (LAN) 46 and the Internet via a communicationscircuit or module 50. LAN 46 interconnects one or more remote computers48, which are configured to communicate with controller 38 using awell-known computer communications protocol such as TCP/IP (TransmissionControl Protocol/Internet(̂) Protocol), RS-232, ModBus, and the like.Additional systems 24 may also be connected to LAN 46 with theprocessors 44 in each of these systems 24 being configured to send andreceive data to and from remote computers 48 and other systems 24. LAN46 may also be connected to the Internet. This connection allowsprocessor 44 to communicate with one or more remote computers 48connected to the Internet. It should be appreciated that communicationsmodule 50 also allows communication between the 3D measurement system 24and a mobile computing device 52 carried by an operator 54. Thecommunications medium between the 3D measurement system 24 and themobile computing device 52 may be via the network 46, or a directwireless connection (e.g. Bluetooth, RFID)

The processor 44 is further coupled to one or more memory devices 56 andan input/output (I/O) controller 58. The memory devices 56 may includeone or more random access memory devices, non-volatile memory devices orread-only memory devices.

It should be appreciated that the mobile platform 26 may includeadditional devices, such as but not limited to lights 60, antennas 62and stop actuators 64 that are mounted to the body 30 as is known in theart. In an embodiment, a post 66 extends from a central area of the topof the body 30. The 3D measurement system 28 is mounted to an end of thepost 66 opposite the body 30. In an embodiment, the post 66 is of afixed length such that the 3D measurement system 28 performs scans at auniform height from the ground. In another embodiment shown in FIG. 5,the post is a scissor lift 68 that is controllable by the processor 44to change the height of the 3D measurement system 28 based on theenvironment in which the system 20 is operating.

In an embodiment, the 3D measurement device 28 is a time-of-flight laserscanner. It should be appreciated that while embodiments disclosedherein may refer to a laser scanner, the claims should not be solimited. In other embodiments, the 3D measurement device 28 may be anydevice capable of measuring 3D coordinates with a desired level ofaccuracy, such as but not limited to a triangulation scanner, astructured light scanner, a photogrammetry system, a videogrammetrysystem or a panoramic camera system for example.

Referring now to FIGS. 6-8, an embodiment of a laser scanner 70 is shownfor optically scanning and measuring the environment surrounding thelaser scanner 70. The laser scanner 70 has a measuring head 72 and abase 74. The measuring head 72 is mounted on the base 74 such that thelaser scanner 70 may be rotated about a vertical axis 76. In oneembodiment, the measuring head 72 includes a gimbal point 78 that is acenter of rotation about the vertical axis 76 and a horizontal axis 80.The measuring head 72 has a rotary mirror 82, which may be rotated aboutthe horizontal axis 80. The rotation about the vertical axis may beabout the center of the base 74. The terms vertical axis and horizontalaxis refer to the scanner in its normal upright position. It is possibleto operate a 3D coordinate measurement device on its side or upsidedown, and so to avoid confusion, the terms azimuth axis and zenith axismay be substituted for the terms vertical axis and horizontal axis,respectively. The term pan axis or standing axis may also be used as analternative to vertical axis.

The measuring head 72 is further provided with an electromagneticradiation emitter, such as light emitter 84, for example, that emits anemitted light beam 86. In one embodiment, the emitted light beam 86 is acoherent light beam such as a laser beam. The laser beam may have awavelength range of approximately 300 to 1600 nanometers, for example790 nanometers, 905 nanometers, 1550 nm, or less than 400 nanometers. Itshould be appreciated that other electromagnetic radiation beams havinggreater or smaller wavelengths may also be used. The emitted light beam30 is amplitude or intensity modulated, for example, with a sinusoidalwaveform or with a rectangular waveform. The emitted light beam 86 isemitted by the light emitter 84 onto the rotary mirror 82, where it isdeflected to the environment. A reflected light beam 88 is reflectedfrom the environment by an object 90. The reflected or scattered lightis intercepted by the rotary mirror 82 and directed into a lightreceiver 92. The directions of the emitted light beam 86 and thereflected light beam 88 result from the angular positions of the rotarymirror 82 and the measuring head 72 about the axes 76, 80. These angularpositions in turn depend on movement generated by corresponding rotarydrives or motors.

Coupled to the light emitter 84 and the light receiver 92 is acontroller 94. The controller 94 determines, for a multitude ofmeasuring points X, a corresponding number of distances d between thelaser scanner 70 and the points X on 90. The distance to a particularpoint X is determined based at least in part on the speed of light inair through which electromagnetic radiation propagates from the deviceto the object point X. In one embodiment the phase shift of modulationin light emitted by the laser scanner 70 and the point X is determinedand evaluated to obtain a measured distance d.

The speed of light in air depends on the properties of the air such asthe air temperature, barometric pressure, relative humidity, andconcentration of carbon dioxide. Such air properties influence the indexof refraction n of the air. The speed of light in air is equal to thespeed of light in vacuum c divided by the index of refraction. In otherwords, c_(air)=c/n. A laser scanner of the type discussed herein isbased on the time-of-flight (TOF) of the light in the air (theround-trip time for the light to travel from the device to the objectand back to the device). Examples of TOF scanners include scanners thatmeasure round trip time using the time interval between emitted andreturning pulses (pulsed TOF scanners), scanners that modulate lightsinusoidally and measure phase shift of the returning light (phase-basedscanners), as well as many other types. A method of measuring distancebased on the time-of-flight of light depends on the speed of light inair and is therefore easily distinguished from methods of measuringdistance based on triangulation. Triangulation-based methods involveprojecting light from a light source along a particular direction andthen intercepting the light on a camera pixel along a particulardirection. By knowing the distance between the camera and the projectorand by matching a projected angle with a received angle, the method oftriangulation enables the distance to the object to be determined basedon one known length and two known angles of a triangle. The method oftriangulation, therefore, does not directly depend on the speed of lightin air.

In one mode of operation, the scanning of the volume around the laserscanner 70 takes place by rotating the rotary mirror 82 relativelyquickly about axis 80 while rotating the measuring head 72 relativelyslowly about axis 76, thereby moving the assembly in a spiral pattern.In an exemplary embodiment, the rotary mirror rotates at a maximum speedof 5820 revolutions per minute. For such a scan, the gimbal point 78defines the origin of the local stationary reference system. The base 74rests in this local stationary reference system.

In addition to measuring a distance d from the gimbal point 78 to anobject point X, the scanner 70 may also collect gray-scale informationrelated to the received optical power (equivalent to the term“brightness.”) The gray-scale value may be determined at least in part,for example, by integration of the bandpass-filtered and amplifiedsignal in the light receiver 92 over a measuring period attributed tothe object point X.

The measuring head 72 may include a display device 98 integrated intothe laser scanner 70. The display device 98 may include a graphicaltouch screen 100, as shown in FIG. 6, which allows the operator to setthe parameters or initiate the operation of the laser scanner 70. Forexample, the screen 100 may have a user interface that allows theoperator to provide measurement instructions to the device, and thescreen may also display measurement results. In an embodiment, thescreen 100 also provides a user interface to allow the operator tointeract, configure and initiate operation of the mobile platform 26 aswell.

In an embodiment, the laser scanner 70 includes a carrying structure 102that provides a frame for the measuring head 72 and a platform forattaching the components of the laser scanner 70. In one embodiment, thecarrying structure 102 is made from a metal such as aluminum. Thecarrying structure 102 includes a traverse member 104 having a pair ofwalls 106, 108 on opposing ends. The walls 106, 108 are parallel to eachother and extend in a direction opposite the base 74. Shells 50, 52 arecoupled to the walls 106, 108 and cover the components of the laserscanner 70. In the exemplary embodiment, the shells 110, 112 are madefrom a plastic material, such as polycarbonate or polyethylene forexample. The shells 110, 112 cooperate with the walls 106, 108 to form ahousing for the laser scanner 70.

On an end of the shells 110, 112 opposite the walls 106, 108 a pair ofyokes 114, 116 are arranged to partially cover the respective shells110, 112. In the exemplary embodiment, the yokes 114, 116 are made froma suitably durable material, such as aluminum for example, that assistsin protecting the shells 110, 112 during transport and operation. Theyokes 114, 116 each includes a first arm portion 118 that is coupled,such as with a fastener for example, to the traverse 104 adjacent thebase 74. The arm portion 118 for each yoke 114, 116 extends from thetraverse 104 obliquely to an outer corner of the respective shell 110,112. From the outer corner of the shell, the yokes 114, 116 extend alongthe side edge of the shell to an opposite outer corner of the shell.Each yoke 114, 116 further includes a second arm portion that extendsobliquely to the walls 106, 108. It should be appreciated that the yokes114, 116 may be coupled to the traverse 102, the walls 106, 108 and theshells 110, 112 at multiple locations.

The pair of yokes 114, 116 cooperate to circumscribe a convex spacewithin which the two shells 110, 112 are arranged. In the exemplaryembodiment, the yokes 114, 116 cooperate to cover all of the outer edgesof the shells 110, 112, while the top and bottom arm portions projectover at least a portion of the top and bottom edges of the shells 110,112. This provides advantages in protecting the shells 110, 112 and themeasuring head 72 from damage during transportation and operation. Inother embodiments, the yokes 114, 116 may include additional features,such as handles to facilitate the carrying of the laser scanner 70 orattachment points for accessories for example.

On top of the traverse 104, a prism 120 is provided. The prism extendsparallel to the walls 106, 108. In the exemplary embodiment, the prism120 is integrally formed as part of the carrying structure 102. In otherembodiments, the prism 120 is a separate component that is coupled tothe traverse 104. When the mirror 78 rotates, during each rotation themirror 78 directs the emitted light beam 86 onto the traverse 104 andthe prism 120. Due to non-linearities in the electronic components, forexample in the light receiver 92, the measured distances d may depend onsignal strength, which may be measured in optical power entering thescanner or optical power entering optical detectors within the lightreceiver 92, for example. In an embodiment, a distance correction isstored in the scanner as a function (possibly a nonlinear function) ofdistance to a measured point and optical power (generally unscaledquantity of light power sometimes referred to as “brightness”) returnedfrom the measured point and sent to an optical detector in the lightreceiver 92. Since the prism 120 is at a known distance from the gimbalpoint 78, the measured optical power level of light reflected by theprism 120 may be used to correct distance measurements for othermeasured points, thereby allowing for compensation to correct for theeffects of environmental variables such as temperature. In the exemplaryembodiment, the resulting correction of distance is performed by thecontroller 94.

In an embodiment, the base 74 is coupled to a swivel assembly (notshown) such as that described in commonly owned U.S. Pat. No. 8,705,012('012), which is incorporated by reference herein. The swivel assemblyis housed within the carrying structure 102 and includes a motor that isconfigured to rotate the measuring head 72 about the axis 76.

In an embodiment, an auxiliary image acquisition device 122 may be adevice that captures and measures a parameter associated with thescanned volume or the scanned object and provides a signal representingthe measured quantities over an image acquisition area. The auxiliaryimage acquisition device 122 may be, but is not limited to, a pyrometer,a thermal imager, an ionizing radiation detector, or a millimeter-wavedetector. In an embodiment, the auxiliary image acquisition device 122is a color camera.

In an embodiment, a central color camera (first image acquisitiondevice) 124 is located internally to the scanner and may have the sameoptical axis as the 3D scanner device. In this embodiment, the firstimage acquisition device 124 is integrated into the measuring head 72and arranged to acquire images along the same optical pathway as emittedlight beam 86 and reflected light beam 88. In this embodiment, the lightfrom the light emitter 84 reflects off a fixed mirror 126 and travels todichroic beam-splitter 128 that reflects the light 130 from the lightemitter 84 onto the rotary mirror 82. The dichroic beam-splitter 128allows light to pass through at wavelengths different than thewavelength of light 130. For example, the light emitter 84 may be a nearinfrared laser light (for example, light at wavelengths of 780 nm or1150 nm), with the dichroic beam-splitter 128 configured to reflect theinfrared laser light while allowing visible light (e.g., wavelengths of400 to 700 nm) to transmit through. In other embodiments, thedetermination of whether the light passes through the beam-splitter 128or is reflected depends on the polarization of the light. The digitalcamera 124 obtains 2D images of the scanned area to capture color datato add to the scanned image. In the case of a built-in color camerahaving an optical axis coincident with that of the 3D scanning device,the direction of the camera view may be easily obtained by simplyadjusting the steering mechanisms of the scanner—for example, byadjusting the azimuth angle about the axis 76 and by steering the mirror82 about the axis 80.

Referring now to FIG. 9, an embodiment is shown of the system 20. Asdiscussed above, the measurement system 24 includes a communicationsmodule 50 that allows the measurement system 24 to transmit to andreceive data from the network 46 via a communications medium 132. Thecommunications medium 132 may be a wireless (e.g. WiFi or radiofrequency communications) or a wired (e.g. Ethernet). It should beappreciated that the communications medium 132 may also allow directcommunications between the mobile computing device 52 and themeasurement system 24, such as a wireless communications protocol suchas Bluetooth™ provided by Bluetooth SIG, Inc for example.

The mobile computing device 52 includes a processor 134 and memory 136.The processor 134 is responsive to executable computer instructions andperforms functions or control methods, such as those illustrated inFIGS. 10-18. The processors may be microprocessors, field programmablegate arrays (FPGAs), digital signal processors (DSPs), and generally anydevice capable of performing computing functions. The one or moreprocessors have access to memory for storing information. The memory 136may include random access memory (RAM) or read-only memory (ROM) forexample, for storing application code that is executed on the processor134 and storing data, such as coordinate data for example. The mobilecomputing device 52 further includes communications circuits, such asnear field communications (ISO 14443) circuit 138, Bluetooth™ (IEEE802.15.1 or its successors) circuit 140 and WiFi (IEEE 802.11) circuit142 for example. The communications circuits 138, 140, 142 aretransceivers, meaning each is capable of transmitting and receivingsignals. It should be appreciated that the cellular phone may includeadditional components and circuits, such as a cellular communicationscircuit, as is known in the art.

In the exemplary embodiment, the mobile computing device 52 includes adisplay 144 that presents a graphical user interface (GUI) 146 to theuser. In one embodiment, the GUI 146 allows the user to view data, suchas measured coordinate data for example, and interact with the mobilecomputing device 52. In one embodiment, the display 144 is a touchscreen device that allows the user to input information and control theoperation of the mobile computing device 52 using their fingers.

In an embodiment, the mobile computing device 52 may include one or morecameras 148. In an embodiment, at least one of the cameras 148 is adepth-camera such as an RGBD type camera which acquires depthinformation in addition to visual information on a per-pixel basis. Thedepth-camera data may be sent to the processor system through wired orwireless communication channels. As will be discussed in more detailherein, the depth-camera 148 may be used to perform a preliminarymapping of the area and object to be scanned. This allows for mapplanning of the route traversed by the measurement device 24.

In an embodiment, the depth-camera 148 may be include one of two types:a central-element depth camera and a triangulation-based depth camera. Acentral-element depth camera uses a single integrated sensor elementcombined with an illumination element to determine distance (“depth”)and angles from the camera to points on an object. One type ofcentral-element depth camera uses a lens combined with a semiconductorchip to measure round-trip time of light travelling from the camera tothe object and back. For example, the Microsoft Xbox™ One manufacturedby Microsoft Corporation of Redmond, Wash. includes a Kinect depthcamera that uses an infrared (IR) light source to illuminate a 640×480pixel photosensitive array. This depth camera is used in parallel with a640×480 pixel RGB camera that measures red, blue, and green colors.Infrared illumination is provided in the IR illuminators adjacent to thelens and IR array. Another example of a central-element depth cameraincludes a lens and a Model PhotonICs 19k-S3 3D chip manufactured by PMDTechnologies of Siegen, Germany may be used in conjunction with an IRlight source. The measurement distance range of this 160×120 pixel chipis scalable based on the camera layout. Many other central-element depthcameras and associated IR sources are available today. Mostcentral-element depth cameras include a modulated light source. Thelight source may use pulse modulation for direct determination ofround-trip travel time. In another embodiment, the light source may usecontinuous wave (CW) modulation with sinusoidal or rectangular waveformsto obtain round-trip travel time based on measured phase shift.

The depth-camera 148 may also be a triangulation-based depth camera. Anexample of such a camera is the Kinect™ of the Microsoft Xbox™ 360manufactured by Microsoft Corporation of Redmond, Wash., which is adifferent Kinect™ than the Kinect™ of the Microsoft Xbox™ One describedherein above. An IR light source on the Kinect™ of the Xbox™ 360projects a pattern of light onto an object, which is imaged by an IRcamera that includes a photosensitive array. The Kinect™ determines acorrespondence between the projected pattern and the image received bythe photosensitive array. It uses this information in a triangulationcalculation to determine the distance to object points in themeasurement volume. This calculation is based partly on the baselinebetween the projector and the IR camera and partly on the camera patternreceived and projector pattern sent out. Unlike the central-elementdepth camera, a triangulation camera cannot be brought arbitrarily closeto the light source (pattern projector) as accuracy is reduced withdecreasing baseline distance.

In an embodiment, the mobile computing device 52 includes aposition/orientation sensor may include inclinometers (accelerometers),gyroscopes, magnetometers, and altimeters. Usually devices that includeone or more of an inclinometer and gyroscope are referred to as aninertial measurement unit (IMU) 150. In some cases, the term IMU 150 isused in a broader sense to include a variety of additional devices thatindicate position and/or orientation — for example, magnetometers thatindicate heading based on changes in magnetic field direction relativeto the earth's magnetic north and altimeters that indicate altitude(height). An example of a widely used altimeter is a pressure sensor. Bycombining readings from a combination of position/orientation sensorswith a fusion algorithm that may include a Kalman filter, relativelyaccurate position and orientation measurements can be obtained usingrelatively low-cost sensor devices.

In an embodiment, the IMU may be used in cooperation with a depthcamera, such as camera 148 to generate a three-dimensional map 152 ofthe environment in which an object 154 is located. To generate thethree-dimensional map 152, the operator walks around the object 152 andacquires a plurality of images with the depth camera 148 of the object152 and the environment in which it is located. Since the distance tosurfaces in the environment may be determined from the data acquired bydepth camera 148 and the movement of the mobile computing device 52 fromthe IMU 150, three-dimensional coordinates of the surfaces in theenvironment and the object 152 may be determined. In an embodiment, thethree-dimensional coordinates are in a first frame of reference. Thethree-dimensional coordinates of the surfaces allows for the displayingof the environment and object on the user interface 146 mobile computingdevice 52 as shown in FIG. 10 and FIG. 11.

The ability to generate a three-dimensional map of the environmentprovides advantages in allowing the user to operate the system 20 on anad hoc basis. In other words, the system 20 may be operated inenvironments and on objects where an a priori or predeterminedinformation is not required, thus the system 20 provides flexibility theinspection of objects. Further, the three-dimensional map 154 providesadvantages in allowing the operator to selectively define locations 156(FIG. 11) in the environment where the operator would like scans bythree-dimensional measurement device 28 to be performed. Thus, theoperator can define the parameters of the scan (location of scans andpath of the mobile measurement system 24) and transfer the parameters tothe mobile measurement system 24. Once the scans are performed by themeasurement system 24, a point cloud of the object 152 is generated.

It should be appreciated that while embodiments herein show the object152 as being scanned by the measurement system 24, this is for exemplarypurposes and the claims should not be so limited. In other embodiments,the 3D measurement device 28 may be mounted to other carriers orplatforms, such as a movable crane or gantry that allows the topsurfaces of the object 152 (e.g. the top of an airplane) to be scannedand a point cloud generated. It should also be appreciated that thesurface analysis described herein does not require the different scansto be registered in a common frame of reference.

Referring now to FIGS. 12-17 an embodiment of performing a surfaceanalysis on the object 152 is shown. Once the point cloud is generated,an inspection module segments the scans into a set of surfaces, such assegment 160 (FIG. 15) for example, based on edges of the objects. In anembodiment, segments of scans are defined with no sharp edges, to avoidbiasing the surface analysis. Once the segments 160 are defined, thesurface is analyzed by extracting small areas of interest 162 (FIG. 13and FIG. 14) of each segment 160. When it is desired to inspect fordents and bumps, the sliding window is shaped like a circle, as shown inFIG. 13. This process is repeated iteratively until the whole segment160 is fully covered.

Next, the inspection module splits the surfaces into smaller geometricalstructures/identities or reference geometries, such as planes, curvedsurfaces, three-dimensional splines, and other shapes for example (e.g.by applying plane fitting to identify the surfaces as shown in FIG. 12).In an embodiment, the plane 164, is a best fit for all of the points inthe area of interest 162. With the plane 164 fit to the area of interest162, a metric, such as but not limited to: the distances between thepoints and the reference geometry, surface normal from the referencegeometry, average/covariance of the distances of window-points to thereference geometry, distribution of the distances (e.g. a region withinthe window with a large number points deeper than a threshold), ratiosof the average/covariance of the distance for two subareas within awindow (e.g. the window is subdivided and the average/covariancedistances are compared, if the values are within a predetermined rangeof each other, then no dent, otherwise the area is marked as a dent),root-mean-square-error (RMSE) values of all the points compared againstthe reference geometry (RMSE greater than a threshold is marked as adent), distribution of normal from the reference geometry, and ananalysis of the extreme points (e.g. maximum or minimum) inside of thewindow. In the exemplary embodiment, the metric is a distance betweeneach point and the plane. In an embodiment, this distance is determinedalong a vector normal to the plane. This allows for the identificationof bumps (e.g. projections) or dents (e.g. recesses) on the surface. Ifthese distances are approximately uniform as is shown in FIG. 14,meaning no peaks are identified, then the surface is marked asdamage-free. However, if there is a variation, a bump or projection 166may be identified. In an embodiment, in addition to the comparison ofthe variation with a pre-determined threshhold, a statistical analysisof all the aforementioned metrics may be performed, and then the area ismarked as having damage. As shown in FIG. 16, these identified bumps orprojections 166 may then be displayed for the operator.

It should be appreciated that the above method may be used forinspection for additional items of interest in addition to bumps anddents. Referring to FIG. 17, an embodiment is shown of a segment 160Ahaving a plurality of fasteners 168. It should be appreciated that anoperator of an object such as an airplane may desire to confirm that allof the fasteners that couple a panel to a body are present. It should beappreciated that where a fastener is missing, a hole will be present inthe segment 160A, such as hole 170. The hole 170 will represent a largedeviation from the plane 164. As a result, embodiments of the disclosedmethod may identify missing parts, misassembled parts and otherstructural elements that deviate from the surrounding areas. It shouldbe appreciated that in an embodiment, the inspection module may identifyrepetitive patterns (e.g. a line of fasteners) and where the pattern isdisrupted, the area may be identified for further inspection. In thisembodiment, the identification of fasteners with a missing head portionmay be identified.

It should be appreciated that this method provides advantages overtraditional methods that used CAD models or create a reference pointcloud for comparison. In some embodiments (e.g. automobile damageinsurance claim inspections), these reference models may not beavailable as they are proprietary to the original manufacturer of theobject. Further, the method of the embodiments described herein alsowork on very large objects (e.g. airplanes or ships) where the referencemodels may not be available or may include deviations that are large dueto build tolerances.

It should be appreciated that while embodiments herein may refer to themobile computing device 52 as being a cellular phone or a tablet typecomputer, this is for exemplary purposes and the claims should not be solimited. In other embodiments, the mobile computing device 52 may be anytype of computing device that is portable and configurable to executethe functions described herein. The mobile computing device 52 may be,but is not limited to a cellular phone, a tablet computer, a laptopcomputer, a personal digital assistant, a special-purpose computer, awearable computer, or a combination thereof.

Referring now to FIG. 18, a method 200 is disclosed for performing asurface inspection of an object. The method 200 starts in block 202where the object 152 or the environment in which the object 152 islocated are mapped. In an embodiment, the mapping is performed byacquiring a plurality of images using the depth camera 148 andmonitoring the position of the mobile computing device 52 with the IMU150. The method 200 then proceeds to block 204 where the map 154 of theobject 152 and the environment are displayed on the user interface 146of the mobile computing device 52. In an embodiment, the object 152 maybe flattened into a two dimensional representation or represented as ashadow over the environment (e.g. the floor). This may occur withobjects, such as airplanes, where the mobile measurement system 24 canmove under the object and to simplify the display for the operator ofthe mobile computing device 52.

In block 204, the operator may indicate locations 156 where scans are tobe performed. In an embodiment, the operator may also indicate a path orroute the mobile measurement system 24 is to travel. The method 200 thenproceeds to block 206 where the scan location data or routing/path dataare transferred to the mobile measurement system 24 and the operation ofthe mobile platform 26 is initiated. The mobile platform 26 moves alonga path to the first location 156. In one embodiment, the mobile platform26 is an autonomous unit that travels along an arbitrary path accordingto predetermined rules. In another embodiment, the mobile platform 26moves along a predetermined path indicated by the operator in block 204.

Once the mobile platform 26 reaches the first location 156, the mobileplatform 26 stops and the method proceeds to block 208. In anembodiment, the wheels 32 of the mobile platform 26 may be located inplace during a scan. In another embodiment, the wheels 32 areretractable. At the location 156, the measurement device 28 initiatesoperation as described herein. The measurement device 28 acquiresthree-dimensional coordinates of points on the surfaces of the object152. It should be appreciated that in an embodiment where themeasurement device 28 is mounted to a scissor lift 68, the height of themeasurement device 28 relative to the floor may be changed prior toinitiating operation. If the operator identified additional locations156 for scans, the mobile platform 26 moves once the scan is completedby the measurement device 28 to the next location. This continues untilall scans have been performed at all locations 156 defined by theoperator.

The method 200 then proceeds to block 210 where the point cloud isgenerated. Next, the surfaces of the object 152 are analyzed in block212. As described herein, the analysis of the surfaces includesdetermining segments 160; determining areas of interest 162 for eachsegment; identifying geometrical structures on the segment; anddetermining the distance from each of the points in the area of interestto the identified geometrical structure (e.g. planes, curved surface,etc.). The method 200 then proceeds to block 214 where areas of interestwith potential anomalies (e.g. bumps and dents) are highlighted on adisplay for the operator. It should be appreciated that the analyzing ofthe object and highlighting anomalies may be performed on themeasurement device 28, the mobile computing device 52, a remotecomputer, a service in the cloud or a combination of the foregoing.

Terms such as processor, controller, computer, DSP, FPGA are understoodin this document to mean a computing device that may be located withinan instrument, distributed in multiple elements throughout aninstrument, or placed external to an instrument.

While the invention has been described in detail in connection with onlya limited number of embodiments, it should be readily understood thatthe invention is not limited to such disclosed embodiments. Rather, theinvention can be modified to incorporate any number of variations,alterations, substitutions or equivalent arrangements not heretoforedescribed, but which are commensurate with the spirit and scope of theinvention. Additionally, while various embodiments of the invention havebeen described, it is to be understood that aspects of the invention mayinclude only some of the described embodiments. Accordingly, theinvention is not to be seen as limited by the foregoing description, butis only limited by the scope of the appended claims.

What is claimed is:
 1. A system for analyzing a surface of an object,the system comprising: a three-dimensional (3D) measurement deviceoperable to acquire a plurality of points on the surface of the objectand determine 3D coordinates for each of the plurality of points; andone or more processors operably coupled to the 3D measurement device,the one or more processors being responsive to executable computerinstructions when executed on the one or more processors for performinga method comprising: generating a point cloud from the 3D coordinates ofthe plurality of points; extracting a first set of points from theplurality of points; defining a first reference geometry through thefirst set of points; measuring at least one first metric from each ofthe points in the first set of points to the first reference geometry;and identifying a nonconforming feature based at least in part on the atleast one first metric.
 2. The system of claim 1, wherein the methodfurther comprise transmitting a signal when the distance exceeds apredetermined threshold.
 3. The system of claim 1, wherein the methodfurther comprises segmenting surfaces in the point cloud into a set ofsurfaces based on edges of the object.
 4. The system of claim 3, whereinthe method further comprises: defining an area of interest, the firstset of points being disposed within the area of interest; and moving thearea of interest over each surface in the set of surfaces, wherein foreach movement of the area of interest a second set of points isextracted from the plurality of points, a second reference geometry isfit through the second set of points, a second metric is measured fromeach of the points in the set of points to the reference geometry, and anonconforming feature is identified based at least in part on the secondmetric.
 5. The system of claim 4, wherein: the first reference geometryis a first plane; and the second reference geometry is a second plane.6. The system of claim 5, wherein: the at least one first metric is afirst distance from the first plane to each of the points in the set ofpoints; and the second metric is a second distance from the second planeto each of the points in the set of points.
 7. The system of claim 6,wherein the first distance is and second distance is determined along avector normal to the first plane and the second plane.
 8. The system ofclaim 1, further comprising a mobile platform, wherein the 3Dmeasurement device is coupled to the mobile platform.
 9. The system ofclaim 8, wherein the mobile platform is operable to autonomously movealong a predetermined path.
 10. The system of claim 8, furthercomprising: a mobile computing device, the mobile computing devicehaving one or more second processors responsive to executable computerinstructions when executed on the one or more second processors, the oneor more second processors being responsive to execute a methodcomprising: displaying a graphical representation of the object on auser interface; identifying locations relative to the object to performsmeasurements with the 3D measurement device in response to an input froman operator; and transmitting the locations to the one or moreprocessors.
 11. The system of claim 10, wherein the mobile computingdevice further includes a 3D camera operable to measure a depth to asurface from the mobile computing device.
 12. The system of claim 11,wherein the one or more second processors are further responsive toexecute a method comprising: acquiring a plurality of images of theobject with the 3D camera; and generating the graphical representationof the object based at least in part on the plurality of images.
 13. Thesystem of claim 1, wherein the nonconforming feature is at least one ofa dent, a bump, or a missing fastener.
 14. A method of analyzing asurface of an object, the method comprising: performing a scan of theobject with a 3D measurement device, the 3D measurement device beingoperable to measure 3D coordinates for a plurality of points on thesurface; generating a point cloud from the 3D coordinates of theplurality of points; extracting a first set of points from the pluralityof points; defining a first reference geometry through the first set ofpoints; measuring at least one first metric from each of the points inthe first set of points to the first reference geometry; and identifyinga nonconforming feature based at least in part on the at least one firstmetric.
 15. The method of claim 14, further comprising segmentingsurfaces in the point cloud into a set of surfaces based on edges of theobject.
 16. The method of claim 15, further comprising: defining an areaof interest, the first set of points being disposed within the area ofinterest; and moving the area of interest over each surface in the setof surfaces, wherein for each movement of the area of interest a secondset of points is extracted from the plurality of points, a secondreference geometry is fit through the second set of points, a secondmetric is measured from each of the points in the set of points to thereference geometry, and a nonconforming feature is identified based atleast in part on the second metric.
 17. The method of claim 16, furthercomprising: moving the 3D measurement device about the object; andperforming a plurality of scans as the 3D measurement device is movedabout the object.
 18. The method of claim 17, wherein the 3D measurementdevice is moved by an autonomous mobile platform.
 19. The method ofclaim 18, further comprising: displaying a graphical representation ofthe object on a user interface; identifying locations to perform each ofthe plurality of scans relative to the object in response to an inputfrom an operator; and transmitting the locations to the autonomousmobile platform.
 20. The method of claim 19, further comprising:acquiring a plurality of images using a 3D camera, each of the pluralityof images including depth information; and generating the graphicalrepresentation of the object based at least in part on the plurality ofimages.
 21. The method of claim 14, wherein the identified thenonconforming feature is at least one of a dent, a bump or a missingfastener.
 22. The method of claim 21, further comprising: displaying thepoint cloud on the user interface; and changing a color of the pointcloud at a location of the nonconforming feature.
 23. The method ofclaim 14, wherein the first reference geometry is a plane, and the atleast one first metric is a distance from the plane to from each of thepoints in the first set of points.
 24. The method of claim 23, whereinthe distance is determined along a vector normal to the plane.